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1.  Introduction 

Let  W(t),  t  >  0,  W(0)  *  0  be  a  standard  Wiener  process,  independent 
of  N(t),  t  >  0,  N(0)  *  Q  a  Poisson  process  with  constant  unit  jumps,  and 
EN(t)  =  Xt,  X  >  0.  Let  their  sigma  fields  be  F(t)  »  ff(W(s),  0  <  s  <  t)  and 
G(t)  =  a(N(s),  0  <  s  <  t).  Let  a  stochastic  process  X(t)  be  defined  in 
terms  of  u(t)  a  u(t,  X(t)),  a  non- anticipating  control,  and  W(t),  N(t) ,  for 
0<t<T,  T>0a  constant,  by  the  equation 

dX(t)  »  u(t)dt  +  dW(t)  +  dN(t)  (1) 

x(0)  *  x,  constant, 

where  u(t)  is  measurable  with  respect  to  cr(F(t)  U  G(t)), 

<i.e.  u  is  non-anticipative) ,  and 
satisfies  for  constants  A,  B  >  0, 

|  u-Aj  <  B  all  0  <  t  <  T.  (2) 

The  cost  function  for  a  given  u  satisfying  (2),  is,  for  cr  >  0- 

,T  . 

J(u)  •  j  e”**^  E (x  (y))dy.  (3) 

J  0 

The  object  of  this  paper  is  to  exhibit  sufficient  conditions  so  that  a 
solution  of  a  resultant  Bellman  equation  yields  an  optimal  admissible  control 
u0(t),  0  <  t  <  T  which  minimizes  (3).  The  sufficient  conditions  are  that 
the  solutions  to  two  homogeneous,  constant  coefficient  partial  differential- 
difference  equations  have  solutions  of  certain  growth,  and  that  the  Bellman 
function  satisfy  certain  matching  and  boundary  conditions. 
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The  method  follows  Ref.  1.  See  also  Ref.  3. 


2.  Finite  Interval  Control 

Lemma  1  Let  D,  \  >  0,  a  >  0  be  constants. 

The  partial  differential-difference  equation  in  V(t,x)  given  by 

2 

x2  +  D  ^  V(x,t)  +~  V(x,t)  -  «V(x,t) 
d  ax 

*-  £  V(x,t)  +  \(V(x+l,t)  -  V(x,t))  =  0  (4) 

O  c 

has  a  particular  solution  expressible  as 

»t  2 

J(D,x,t)  a  (  e^ECDy  +  N(y)  +  W(y)  +  x)  dy  = 
fc0 


—  (l-e”0ft)  +  )  (X  +  1  +  2Dx  +  2),x) 

a 

+  «L±-u!  (2 . .  2e-« . 

a 

where  N(y),  W(y)  are  as  in  section  l. 

The  differential-difference  equation  in  V(x)  given  by 

x2  +  DV'(x)  +  |  V"(x)  -  orV(x)  +  X (V (x+1)  -  V(x))  - 
has  a  particular  solution  expressible  as 

j(D,x)  a  f  e~°^E (Dy  +  N(y)  +  W(y)  +  x)2dy  , 

*  r\ 


(5) 


(6) 


i!  +  X-H+2Dxf2^x  +  2  (D  +  X)2  (7) 

a  a2  *3 

proof  The  proofs  are  direct  computation  upon  expansion  and 
evaluation  of  (5),  (7)  respectively. 

Remark:  The  solutions  represent  the  respective  costs,  if  the  constant  control 
u(t)  a  D  is  employed. 

Theorem  1.  Let  X(t)  be  given,  for  0  <  t  <  T,  by 


dX(t)  «  udt  +  dN(t)  +  dW(t) 
x(0)  ■  x  with  assumptions  of  section  l. 
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* 


vith  cost  function  *  for  0  <Tl<»  constant, 

T 

J(u)  -  r  e'“yE(X2(y))dy. 

J  0 

The  optimal  control  ug(t)  which  satisfies 

|uQ(t)  -  A|  <  B  0  <  t  <  T 

is  given  by 


A-B  if  XQ(t)  >  b(T-t) 


u  (t)  = 
o 


A+B 


if  X0(t)  <  b(T-t) 


(8) 


where 

dXQ(t)  -  uQdt  +  dN(t)  +  dW(t) 

and  it  is  assumed  that  b(t)  satisfies  transcendental  equations  (131,  (20)- (21), 
given  below. 

Proof .  The  Bellman  equation  for 


V(t,x)  a  inf  f  »"®yvrv2 

}u-Aj  <  B  *  0 


e  ar7E(X"(y))dy 


(9) 


with  X(0)  *  x 


is  seen  by  heuristics  or  from  Ref.  2  pp.  179-180  to  be.  where  now 
2 

V  m  V  ,  -3-r  V  a  V  ,  etc., 

Ax  x’  2  xx  ’ 

ax 


X2  +  inf  (uv  (x,t))  +  |  V  (x.t)  -  arV(x,t) 

J  u-Aj  <  B  *  1  ** 


-  Vc(x,t)  +  \(V(x+l,  t)  -  V(x,t))  -  0.  (10) 


Intuitive  considerations  suggest  that  a  function  b(t)  be  sought  such 
that 
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V^x.t)  satisfies 


x2  +  (A-B)  V  (x,t)  +  i  V  (x,t)  -  «V(x,t)  -  V  (x,t) 

X  &  XX  » 

+  x (y (x+i , t>  -  v(x,t>)  -  o  (id 

if  Vx(x,t)  >  0  and  x  >  b(t), 

and  V2(x,t)  satisfies 

x2  +  (A+B)  vx(x,t)  +  \  Vxx(x,t)  -  afV(x,t)  -  Vt(x,t) 

+  X(V(x+l,t)  -  V(x,t))  -  0  (12) 

if  Vx(x,t)  <  0,  and  x  <  b(t). 

The  boundary  conditions  are,  for  0  <  t  <  T, 

V^x.O)  -  V2(x,0)  -  0  all  x, 

^Vi(b(C).o  .^v2(b(t),t)  .0 

V1(b(t),t)  -  V2(b(t),t)  (13) 

By  Lemma  1,  J(A-B,x,t),  J(A+B,x,t)  are  particular  solutions  of  (11), 

(12)  respectively. 

Assumption  1.  There  is  a  non-zero  solution  H^(x,t)  to  (omitting 
(x,t)  arguments) 

(A-B)Hx  +  -»H  -Ht  +  X(H(x+l,t)-H)  -  0  (14) 

such  that 

Ht (x,0)  -  0  (15) 

Hj (x,t,y)  -  0(e-Px) 

H.  "  0(e'6X)  <l6) 

l  ,xx 


4 


,  „  ^  r»  e  >  o.  each  C,  as  x  -»  ». 

for  some  p  >  u>  o  *  u* 

Also,  there  is  a  non- zero  solution  »2(x,t,y)  with 


H2(x,0)  *  0 


to 


such  that 


-«H  -»c  +  X(H<«+1.«-a>  *  0 


h2<jc,c>  '  0(«YX) 


for  some  V  >  0,  A  >  0,  each  t,  as 


Then  one  has 


V^x.t)  »  J(A-B,x,t)  +  H^Cx.t) 
V2(x,t)  *  JCA  +  B,x,t)  +  H2(x,t) 
and  b(t)  is  determined  by  (13),  (20)-(21). 


Lemma  2 


V  (x,t) 

xx 


vi  «<*»*>  >  °»  *  >  b(C) 

V_  (x,t)  >0,  x  <  b(t) 
2, xx 


(17) 

(18) 

(19) 


(20) 

(21) 


(22) 

(23) 
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Proof  Let  W(x,t)  =  V^Cx.t). 

Then  from  (11),  (12), 

(A-B)  Mx(x,t)  +  Wxx(x,t)  -  (or+xJ  W(x,t) 

-  Wt(x,t)  -  -2  -  \W(x+l,t)  (24) 

x  >  b  (t) 

(A+B)  Wx(x,t)  +|wxx(x,t)  -  (a+X)  W(x,t) 

-  Wt(x,t)  -  -2  -  xW(x+l,t)  (25) 

x<  b(t) 

By  construction  of  V^x.t),  V2(x,t)  in  (14)-(21),  W(x,t)  ■  Vxx(x,t)  >  0 
for  each  t,  for  all  x  sufficiently  large. 

Suppose  there  is  an  x^  finite,  possibly  depending  on  t,  such  that 
W(xQ,t)  <  0,  and  W(x,t)  >  0,  x  >  xQ- 

Then  the  left  sides  of  (24),  (25)  are  negative  for  x  >  x^  -1.  By  ref. 

4,  Lima?  ,  p.  34,  W(x,t)  cannot  have  a  negative  minimum  for  x  >  Xq  -  1.  But 
since  if  W(xQ,t)  <  0,  and  W(x,t)  >  0,  x  >  xQ,  W(x,t)  >  0,  x  -*  then  W 
would  have  a  negative  minimum  for  x  >  x^  -  1,  a  contradiction  to  the 
existence  of  Xg,  hence  Lemma  2  is  proved. 
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Upon 


taking  expectations  in  (28),  one  obtains 


f  e‘“yE(X2(y))dy  -  V(x,T)  -  • 

“0 

EfT.-"(-»v(x(y)  ,y)  -  vt«(y).y)  +  ,*!£,<g<“<X<1'>  •y)Vx<y) 
*0  'u  — 

+  X2(y)  +  \  + 

+  \(V(X(y)  +  1,  y)  -  V(X(y),y))dy 


(29) 


Th*  integral  on  the  right  side  of  <W  is  rero  hy  (10),  end 

che  second  integral  on  the  right  is  n.n-neg.tive ,  with  egnalit,  for  n  -  n 

Hence 

T  e“ayE(X2(y))dy  >  V(x,l) 

1  0 

for  any  admissible  u,  and 

-T  .aY  2 . 

!  ayE  (> 

0 


f  e'ayE(X2(y))dy  -  V(x,T), 
*n  0 


(30) 


(31) 


for  u  -  uQ>  so  that  uQ  is  optimal. 

Remark:  There  is  no  claim  that  uQ  above  is  unique. 


3.  Infinite  Interval  Control 


Theorem  2.  Let  X(t)  be  given  by 


dX(t)  -  udt  +  dN(t)  +  dW(t) 

for  all  t  >  0,  and  X(0)  -  x,  satisfying  the  assumptions  of  section  1 


vich  cost  function 


(32) 


J(u)  =*  f  e'“yE(X2(y))dy. 

*  0 

The  optimal  control  uQ(t)  which  satisfies 
[uQ(t)  -  A!  <  B  all  t  >  0  is 

given  by 

[A  -  B  for  Xx(t)  >  b 

A  +  B  for  X1(t)  .<  b  (33) 

where 

dX1(t)  «  ^dt  +  dN(t)  +  dW(t), 

and  ic  is  assumed  chat  b  is  a  constant  which  satisfies  trancendental  relations 
(38)- (42). 

Proof  The  Bellman  equation  for 

V(x)  a  inf  fVoyE(X2(y))dy  (34) 

»u-A[<B  J0 

with  X(0)  »x  is  (Ref.  2,  pp.  179-180) 

x2  +  inf  (uV1  (x))  +~  V"(x)  -  QfV(x)  +  X(V(x+l)-V(x))  -  0.  (35) 

|  u-Al<B 

A  solution  of  the  following  form  is  sought.  V^x)  satisfies 

x2  +  (A-B)V’(x)  +  -j  V”(x)  -  aV(x)  +  X(V(x+-l)-V(x))  -  0.  (36) 

for  V’(x)  >0,  x  >  b 
and  V2(x)  satisfies 

x2  +  (A+B)V’(x)  +  |  V"(x)  -  aV(x)  +  X(V(x+l)-V(x))  =■  0  (37) 
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where 


V' (x)  <  0,  x  <  b. 


The  matching  conditions  are 
V1*(b)  =  V2'(b)  =  0 

VL(b)  =  V2(b).  <38> 

By  Letnma  1,  J(A-B.x)  is  a  particular  solution  to  (36)  and  J(A+B,x) 
is  a  particular  solution  to  (37).  A  solution  to  the  homogeneous  parts 
of  (36) ,(37)  is  obtained  as  follows:  let 

f (r)  =  r2  +  2(A-B)r  -  2(a  +  X)  +  2Xer  (39) 

and 

k(r)  =  r2  +  2 (A+B) r  -  2(a  +  \)  +  2\eT.  (40) 

Since  f(0)  *  k(0)  =  -2 a  <  0,  and  f(-o)  =  +  =  ,  k(“)  =  +“  , 
there  exist  r^  <  0,  r^  >  0  such  that  f(~^)  =  k(r^)  *  0. 

Hence  a  solution  to  (36)  is 

Vl(x)  =  J(A-B,x)  +  CerIX  (41) 

for  x  >  b 

r?x  (42) 

and  V2(x)  =  J (A+B ,x)  +  De  L 

for  x  <  b, 

and  it  is  assumed  that  constants  C,D,b  are  determined  by  conditions  (38).  It  is 
required  to  show  that  (41),  (42)  solve  the  Bellman  equation-,  that  Is, 
that  (36) ,(37)  hold. 
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Proof  Since  V^'(b)  =  V^'Cb)  =*  0,  it  suffices  to  show  that 
V^'(x)  >  0  for  x  >  b  and  V2’(x)  <  0  for  x  <  b.  For  this  it  suffices  to 
show  that  V"(x)  >  0  all  x  £  b.  Let  w(x)  =  V"(x)  and  from  (36), (37),  one 
obtains  that 

(A-B)w’(x)  + -|  w"(x)  -  (or  +  X)w(x)  *  -2  -  Xw(x+1).  (43) 

and  for  x  >  b 

(A+B)w'(x)  +  \  w"(x)  -(«  +  X)w(x)  =  -2  -Xw(x+1)  (44) 

for  x  <  b. 

By  construction  of  the  solution  in  (39)- (42), 
w(x)  >  0  for1  all  x  sufficiently  large. 

Suppose  there  is  an  x^  such  that  w(x^)  <  0  and  w(x)  >  0,  x  >  x^. 

Then  the  right  sides  of  (43)  ,  (44)  are  negative  for  x>  xQ-l,  and  hence  the 
left  sides  of  (43),(44)  are  negative  for  x  >  xQ-l.  By  Ref.  4,  p.  53,  Theorem 
19,  w(x)  cannot  have  a  negative  minimum  for  x  >  Xq-1,  a  contradiction  to 
the  existence  of  x  ^  such  that  w(x^)  <  0.  This  suffices  to  prove  Lemma  4. 

To  complete  the  proof  of  Theorem  2  it  remains  to  show  that  u^(t)  of  (33)  is 
optimal. 

Lemma  5  u^t)  is  optimal. 

Proof  For  a  fixed  u,  |u-A]  <  B,  let 


dX(t)  =  u(t)dt  +  dW(t)  +  dN(t) 

X(0)  =  x 
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and  define 


H(X(t),c)  =  V(X(t))e 


-art 


V1(X(t))e‘0(t,  X(t)  >  b 


V2(X(t))e'at,  X(c)  <  b. 


(45) 


Noting. that  H(X(0),0)  =  V(x),  an  application  of  Iro’s  formula  (Ref.  2,  p. 
yields,  upon  subsequent  integration  from  0  Co  t, 


f  e'oy<X2(y))dy  +  e~atV(X(t))-V(x)  = 

J0 

f  e'flry(-aV(X(y))  +  u(X(y))Vf  (X (y) )  +  X2(y)  V"(X(y))dy 
An  * 


f  e'0fy(V(X(y))cjN{^ 


A  * 

+  j  e'®yV,(X(y))dW(y), 
J0 


Upon  taking  expectations  in  (46),  one  obtains 


f  e_aryE (X2(y) )dy  -  e*® CEV(X( t) ) -V(x) 
v  0 


(46) 


Ef  e'ay(-aV(X(y)  +  inf  u(X(y) ) V' (X(y))  +  X2(y) 
J0  lu-Al<B 


+  \  V"(X(y))  +X(V(X(y)+l)  -  V(X(y)))dy 
r t 

+E|  e""y(u(X(y))V'(X(y))  -  inf  u(X(y)) V’ (X(y))dy  (47). 

*  0  ] u-A]<B 

The  first  term  on  the  right  of  (47)  is  zero  by  definition  of  V  in 
(35)- (37).  The  second  term  on  the  right  is  non-r.egacive . 

By  construction  of  V(x)  in  (39)- (42),  for  t  large, 

EV(X(t))  <  KE(x  +  N(t)  +  tW  (t)|  +  (>A|+B)t)2  <  Mt2,  (48) 
for  suitable  constants  K,  M. 


126) 
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